Skip to content

Release: NEMAR MCP server (epic #1065) - #1334

Merged
neuromechanist merged 19 commits into
mainfrom
dev
Sep 9, 2026
Merged

neuromechanist merged 19 commits into
mainfrom
dev

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Promotes the NEMAR Model Context Protocol (MCP) server to production. Opened
with the -devN suffix intact, per the release pipeline: CI owns the strip, the
tag and the npm publish.

What this release is

19 commits, 105 files, +21,118 lines, and almost all of it is one epic. The MCP
server runs inside the existing backend Worker and answers at mcp.nemar.org,
anonymous like the rest of the data plane. Six tools, cheapest first:
search_datasets, describe_dataset, list_recordings, get_events,
render_overview, read_window. The posture is ADR 0049's: a stateless,
recipe-first broker that hands a client what it needs to fetch bytes itself and
decodes signal data only for an opt-in, hard-capped taste.

Plus ADR 0049 (#1292) and ADR 0050 (no WebAssembly in the Worker bundle).

No migrations, so migrations:d1-check does not apply.

Why the deploy matters beyond the code

mcp.nemar.org is a custom_domain = true route, so the hostname does not
exist until this deploys
. That is also why the four recording-level tools are
unverified so far: mcp-test.nemar.org came up on the dev deploy and every
transport-level check passed against it, but zarr-test.nemar.org's catalog is
count: 0 -- the dev environment has no converted dataset at all -- so
list_recordings, get_events, render_overview and read_window have nothing
to read there. Production has 625. Verifying them, and writing the docs section
and OSA's client runtime from those real responses, is what happens immediately
after this merges (issue #1297; #1065 stays open until then).

Verified on staging before promoting

Against the live mcp-test.nemar.org, from the official Python MCP SDK:

  • Streamable HTTP connects; protocol 2026-07-28 negotiated.
  • All six tools present, each with an inputSchema; tools/list carries its
    cache hint (ttlMs=86400000, scope=public).
  • GET /mcp 405, DELETE /mcp 405, OPTIONS /mcp 204, descriptor GET / 200,
    a POST with no Mcp-Method 400, a disallowed Origin 403.
  • describe_dataset end to end on an exemplar, with its sandbox DOI, composed
    citation, and a cost_hint that says conversion has not started.
  • list_recordings on an unconverted dataset refuses with the actual
    zarr_status and points at describe_dataset, rather than answering an empty
    list.
  • The D1 gate, on the live host: nm000103, a real private dataset, is
    refused with the same wording an unknown id gets. No existence leak.

Reviewed as its own diff

The five phase PRs were each reviewed, and then the integrated diff was reviewed
again on #1332 specifically for the seams, which found five criticals. All are
fixed here; the two worth knowing:

  • render_overview threw a RangeError on its default path for much of the
    catalog. Zarr stores a boundary chunk full size and fill-padded, and the
    reassembler inferred each chunk's width from its length. Same nominal-stride
    versus valid-span trap phase 4 fixed for level 0, one axis over -- and the
    fixture encoded the same wrong assumption, which is why the suite was green.
  • The projection cache keyed on source_commit alone, on the false premise
    that a re-conversion always changes it. An engine bump re-converts at an
    unchanged HEAD (ADR 0033), so stale scale[]/offset[] could be served
    against re-quantized data -- silently wrong physical values -- and stale shard
    byte offsets replayed as Range reads where every 206 and length check passes on
    the wrong bytes.

Also: /mcp fell into the token rate-limit bucket on a surface with no auth, two
anonymous reads were unbounded (a 95 MB parquet; a taste that decoded 31 MB to
return 500 KB), and the zod3/zod4 drift test was passing vacuously on phase 4's
entire taste contract. Full write-up in #1332's review comment.

Dependency note, checked rather than assumed: zod 3 stays on its major
(3.25.76). zod 4 arrives only as a zod4 npm alias for the SDK's registerTool,
so no existing route's validation changed. The z.record(z.string(), ...) edits
in admin/doi.ts are the equivalent two-argument form.

Gates

Epic-branch CI green with the live tier actually running (unit-pure,
api-test, integration-dev, e2e-sandbox, zenodo-sandbox-test), 342 MCP
tests, backend/scripts/mcp-smoke.sh green under real workerd (mandatory per ADR
0050), typecheck and biome clean.

neuromechanist and others added 19 commits September 8, 2026 19:32
* chore: bump backend hono to ^4.11.4

Prerequisite for the MCP SDK's @modelcontextprotocol/hono peer
(^4.11.4); previously pinned ^4.6.0. typecheck and the backend
suite are green at this version.

* fix: z.record requires an explicit key schema

The single-argument z.record(valueSchema) form is invalid under
zod 4 (z.record(keySchema, valueSchema) is required); the
two-argument form is valid under zod 3 too, so this is a safe
fix regardless of which major version the repo ends up on.
Found while gating the phase 1 MCP contracts (issue #1293)
against a zod 4 bump.

* test: add live zarr v3 index and catalog fixtures

Captured with curl -sL -A "nemar-cli/mcp-phase1" (issue #1293):
- zarr-index-nm000329-slice.json: first 3 stores of the live
  index, keeping events_parquet and top-level doi/license/citation.
- zarr-index-on003392-meg-sss-slice.json: the one live MEG store
  carrying an ADR 0028 sss object.
- zarr-catalog-slice.json: 3 entries from zarr.nemar.org/catalog.json.
- zarr-array-level0.zarr.json / zarr-array-view1.zarr.json: the
  level-0 and view/1 array metadata documents for nm000329's
  eeg_250hz group, matching the epic's verified geometry.

* feat: zarr index v3 and MCP server contracts

Issue #1293, phase 1 of epic #1065. shared/contract/zarr-index.ts
is a .passthrough() zod reader mirroring shared/zarr-index.schema.json
(the producer's closed contract stays the JSON Schema; this is a
client-side lower bound on it, matching dataset.ts's convention).
shared/contract/mcp.ts is the MCP wire vocabulary ADR 0049 fixes:
the provenance envelope every tool response carries, buildReadRecipe()
(recipe-first read_window, per the plan), and one input/output schema
pair per tool (search_datasets, describe_dataset, list_recordings,
get_events, render_overview, read_window), including the taste cap
(60s x 64 channels) read_window's schema enforces.

Both contract-*.unit.test.ts files exercise the public schema/function
surface against real fixtures (on008083, and the two live slices),
including three deliberate mutation tests (dropped source_commit,
format_version 1, non-hex commit) proving the zod reader rejects
what the Ajv-compiled JSON Schema also rejects.

* feat: MCP transport and decode-path spike

Issue #1293 step 5. Answers the two things the phase 1 plan left
open: does @modelcontextprotocol/server@2.0.0 + hono@2.0.0 serve
both protocol eras under workerd (wrangler dev), and which blosc/
zstd decode path (decision 7) actually runs there.

Both eras of the transport work: server/discover, tools/list,
tools/call in the 2026-07-28 envelope, the legacy initialize
handshake, a legacy tools/call with no envelope, the -32020
rejection (disagreeing Mcp-Name, missing Mcp-Method), and GET/
DELETE 405 all behave as documented -- see smoke.sh.

numcodecs' WASM Blosc codec does not run under workerd (dynamic
WebAssembly.instantiate() on a runtime-fetched buffer is disallowed
by the embedder); the pure-JS decoder in decode.ts (blosc2 header
parse + fzstd + unshuffle) works and matches a Python-derived
ground truth exactly. Path (b) is the chosen decode primitive.
fixtures/chunk.bin is one real inner chunk of nm000329's level-0
array, captured via two range requests against the shard's
trailing index; README.md documents the byte-level derivation and
the full measurement set (bundle size, wall time).

Replaced by phase 2 (#1294) and deleted in that PR.

* docs: MCP server design doc

Issue #1293. Purpose and constraints, the 2026-07-28 protocol
revision and dual-era behavior (with the exact SEP-2243 header/
envelope mechanics verified against a real wrangler dev run, not
just the spec summary), host and routing (phase 2 scope), the six-
tool surface with cost class and cache behavior per tool, the
envelope and read-recipe field tables, compute-minimization rules
and the cache key scheme, the analytics event shape, the pinned
dependency table with the spike's verdicts, the client compatibility
table, and open items for phases 2-5.

ADR 0050 is omitted from this PR per decision 11: origin/dev does
not yet carry ADR 0049 (PR #1292 is still open). The decision text
lives in this document instead; phase 5 files the ADR once #1049
has landed.

* docs: cache the rendered PNG, reject over-cap taste

The design doc drifted from plan decision 4 (cache the rendered overview
PNG keyed by width, not the decoded level) and from its own schema (an
over-cap taste is rejected by the input schema, not downgraded to a
recipe). The zarrita snippet now uses the documented FetchStore form;
zarrita's open.v3 takes a store, not a URL string.

* fix: guard the spike decoder and assert HTTP status in smoke.sh

decodePathB now refuses a short buffer, a non-int16 typesize, a zero
blocksize (blosc2 marks a single block with blocksize == nbytes, never 0),
a cbytes that disagrees with the input, offsets or lengths past the end of
the chunk, and an odd decoded byte length that Int16Array would silently
floor. Offsets and lengths are read unsigned. smoke.sh checks the HTTP
status on every call, keeps bodies in a mktemp dir it removes on exit, and
kills workerd along with wrangler. Smoke run: 10/10 PASS.

* fix: contract review findings for the MCP wire schemas

- a catalog row's null doi/license is authoritative over the index's
  stale hoisted copy; the index is consulted only without a row
- fillTemplate fills layout templates in one pass and throws on a
  template that lacks a placeholder the caller needs
- buildReadRecipe takes Zarr's own array metadata (data_type), with a
  zarrArrayMetadataSchema reader for the served arrays' zarr.json
- a taste requires channels: the schema cannot know n_channels and a
  live MEG store has 320
- sss is present exactly when derived is true, enforced on the store
  reader and the envelope; sss models the converter's known keys
- ranges enforce end >= start; flagToBoolean is the one 0|1|null to
  boolean conversion; min_participants dropped (no server-side filter)
- tests: 26 -> 51, covering every tool input default and cap, both
  read_window output branches, the catalog document, the real
  zarr.json fixtures, and each fallback branch of the envelope

* docs: align the design doc with the reviewed contracts

search_datasets loses min_participants and names the two catalog
routes it wraps; read_window's taste requires channels; the envelope
table states the catalog-over-index rule; the recipe table splits
dtype from codecs; section 12 records how phase 2 registers tools
across the zod 3 / zod 4 split (zod4 npm alias plus a JSON Schema drift
test).
* mcp: dependencies and bindings for the phase 2 host (#1294)

Adds @modelcontextprotocol/server (pinned 2.0.0), the zod4 npm alias
(npm:zod@^4.2.0, resolved by Bun without touching the repo zod pin),
and fzstd (needed by the decoder promotion in the next commit). New
ANALYTICS_MCP and MCP_HOSTNAME bindings in Bindings and
wrangler-sccn.toml (prod + env.dev), matching the DATA_HOSTNAME/
ZARR_HOSTNAME/ANALYTICS_LLM pattern.

* mcp: add the mcp host-route fork and workers.dev path mount (#1294)

New HostRoute arm "mcp" in resolveHostRoute, alongside data/zarr/api,
keyed off MCP_HOSTNAME (default mcp.nemar.org). index.ts forks to the
mcp sub-app on that host and forwards the api host's exact /mcp path
to it too, for the workers.dev/dev fallback -- a single-path forward,
not a full sub-tree mount, so only the transport endpoint (not the
descriptor at "/") is reachable that way.

Note: backend/src/routes/mcp.ts (mcpRoutes) lands in a later commit;
this commit's index.ts import is completed by that one.

* mcp: promote the blosc/zstd decoder out of the phase 1 spike (#1294)

Moves the spike's decodePathB (backend/spike/mcp-transport/src/decode.ts,
deleted in an earlier commit) to backend/src/services/blosc-decode.ts as
decodeBloscZstdInt16, dropping path (a) (numcodecs' WASM Blosc codec,
which the spike proved does not run under workerd) and its import
entirely. Fixtures moved to backend/test/fixtures/blosc/. Not yet wired
into any tool or route -- read_window's taste mode (phase 4) is its
first caller.

* mcp: add composeCitation and describe_dataset's additive fields (#1294)

composeCitation in shared/contract/mcp.ts is a line-for-line TypeScript
port of dataset_citation in scripts/zarr/generate_zarr.py; verified
byte-identical against the nm000329 index fixture's citation, produced
by the Python function from the same public dataset row (captured as
test/fixtures/dataset-row-nm000329.json).

describeDatasetOutputSchema gains zarr_source_commit and
zarr_store_count (additive, optional). provenanceEnvelopeSchema's
module doc is corrected: the envelope rides recording-level tool
responses (phases 3-4), not search_datasets/describe_dataset, which
carry doi/license/citation/zarr_status/zarr_source_commit/
zarr_verify_status directly instead.

* mcp: add the ANALYTICS_MCP metrics service and its tests (#1294)

buildMcpDataPoint/recordMcpToolCall (backend/src/services/mcp-metrics.ts)
mirror buildAccessDataPoint's index+blobs+doubles shape but write to a
dedicated Analytics Engine dataset (ANALYTICS_MCP) so blob positions
never collide with access-metrics.ts's. tool is the index/blob1; the
dataset id (or "-") is blob2; cache_status is blob3; elapsed_ms and
upstream_bytes are the two doubles. No-ops without the binding, never
throws.

The route-level "one point per tools/call, including on isError"
assertion in this test file depends on backend/src/routes/mcp.ts,
which lands in a later commit.

* mcp: add zod4 registration mirrors and the schema parity test (#1294)

backend/src/mcp/schemas.ts mirrors search_datasets'/describe_dataset's
input/output schemas against the zod4 alias, since registerTool needs
a Standard Schema with JSON Schema support (zod 4 has it, the repo's
zod 3.25.76 does not). Same field names, defaults, caps and regexes as
the zod 3 wire contract (shared/contract/mcp.ts), every field
.describe()'d so the published JSON Schema teaches the caller.

mcp-schema-parity.test.ts is the drift test: one shared table of valid
and invalid inputs (defaults, the cap, cap+1, a malformed id, an
unknown passthrough key) run through both copies' .safeParse, asserting
identical accept/reject verdicts and parsed values -- behavior parity,
not a comparison of two different JSON Schema generators' output.

* mcp: add search_datasets, describe_dataset, and the mcp sub-app (#1294)

search_datasets (backend/src/mcp/tools/search-datasets.ts) wraps
executeDatasetSearch when a query is given, or the public-catalog list
query otherwise, filling license/zarr facts SearchResult doesn't carry
via one follow-up query keyed by the page's ids. describe_dataset
(describe-dataset.ts) is one fresh D1 query on the public predicate,
never index.json; unknown/private/sandbox ids answer a tool error
naming search_datasets.

backend/src/mcp/server.ts builds the McpServer (name "nemar", a 24h
public cache hint on tools/list and server/discover) and wraps every
tool callback in withToolMetrics, which records exactly one
ANALYTICS_MCP point per call on both the success and error paths.

backend/src/routes/mcp.ts is the sub-app: its own origin gate
(allowedOrigin/corsHeaders reused from zarr-data.ts, since this server
does not mount @modelcontextprotocol/hono's standalone origin
validation) and rate-limiter bridge, a GET / descriptor, and
createMcpHandler built per request on /mcp.

mcp-route.test.ts drives the full protocol/tool surface against a real
bun:sqlite D1 (both protocol eras, tools/list cache hints, filters,
FTS, tool errors, the Mcp-Name -32020 mismatch, the origin gate, 405/
204). mcp-fork.test.ts drives the real worker entry point
(backend/src/index.ts's default export) for the host-fork and
path-mount wiring.

* mcp: add the workerd smoke script (#1294)

backend/scripts/mcp-smoke.sh starts wrangler dev --local against the
api-host /mcp path mount (the hostname fork can't be exercised on
loopback) and asserts server/discover, tools/list with its cache hint,
a tool error naming search_datasets, a malformed dataset_id's SDK
input-schema rejection, the legacy initialize + tools/call idiom, the
Mcp-Name -32020 mismatch, GET/DELETE /mcp 405, OPTIONS /mcp 204, and
the origin gate.

Could not be run to completion in this environment: wrangler dev
--local fails to start the worker at all (a pre-existing, unrelated
export-shape issue in the unmodified index.ts -- see this phase's PR
body and .context/mcp-server-design.md section 12 for the full
derivation). The script is written and verified equivalent-behavior
via mcp-route.test.ts driving the real SDK under Bun.

* mcp: update the design doc and systems inventory for phase 2 (#1294)

mcp-server-design.md: section 2 corrected (no @modelcontextprotocol/hono
-- this server lives inside the existing single worker behind
resolveHostRoute, not standalone; origin gate and rate limiting done in
the sub-app instead), section 2.1 gains the verified isError-vs-JSON-RPC-
error tool-validation shape, section 3 marked implemented, section 5
corrected (the provenance envelope rides recording-level tools only),
section 8 resolved (a dedicated ANALYTICS_MCP dataset), section 10 gains
the real bundle delta (+1054.40 KiB / +206.36 KiB gzip) and the fixture
relocation note, section 12 phase 2 items ticked, including the
wrangler-dev-local blocker with full reproduction evidence.

systems-inventory.md section 1 gains the mcp.nemar.org/mcp-test.nemar.org
hosts next to data/zarr.

* mcp: schemas.ts imports the limit constants instead of redefining them (#1323 review)

SEARCH_DATASETS_DEFAULT_LIMIT/SEARCH_DATASETS_MAX_LIMIT were redefined
locally in the zod4 mirror file; a mirror must never own a number the
zod 3 wire contract (shared/contract/mcp.ts) already owns. Imports them
from there instead. No other duplicated constant or regex found in the
file -- DATASET_ID_RE was already imported from zarr-index.ts, and the
enum literals (zarr_status, zarr_verify_status, next_cheapest_tool)
have no separately-exported array in the contract to import.

* mcp: describe_dataset uses toVersionTag directly, drops the route import (#1323 review)

Replaces withCanonicalLatestVersion (routes/datasets/catalog.ts) --
which needed a double cast to apply to DescribeDatasetRow anyway --
with toVersionTag (shared/contract/version.ts) applied inline to
row.latest_version when it is a non-empty string, exactly what
withCanonicalLatestVersion itself does. A tool no longer imports a
route module. No test changes needed; mcp-route.test.ts's citation
assertion (composeCitation against the v-tagged version) still passes
unchanged.

* mcp: unblock the workerd smoke with a minimal entry, not index.ts (#1323 review, issue #1324)

wrangler dev --local cannot start backend/src/index.ts in this
environment at all (filed as #1324; out of this phase's scope). Adds
mcp-smoke-entry.ts (re-exports only mcpRoutes.fetch as the default
ExportedHandler) and mcp-smoke.wrangler.toml (same compatibility_date/
compatibility_flags as wrangler-sccn.toml; a local-only DB binding;
ENVIRONMENT=development; nothing else) as a throwaway config the smoke
script points at instead.

Because the entry IS the mcp sub-app, every path -- including "/" --
now reaches it directly, so GET / asserts the real mcp descriptor
instead of the (unrelated) API root the prior path-mount workaround
was limited to.

describe_dataset/search_datasets need the MIGRATED schema even for a
row that doesn't exist (an unmigrated D1 has no `datasets` table,
which surfaces as a D1_ERROR that doesn't name search_datasets --
correcting the "none should [need migrations]" assumption with
evidence). The script now wipes and re-applies every migration to the
local D1 first, via `wrangler d1 execute --local --file` per file with
full-line comments stripped -- the same technique
scripts/d1-migration-check.ts already uses to work around
`d1 migrations apply` false-positiving on "BEGIN TRANSACTION"/"COMMIT"
inside a `--` comment.

Result: all 14 checks PASS on repeated runs, under real workerd, with
the unchanged compatibility_date 2024-12-01 (not bumped). Per-call
wall times and the full PASS/FAIL table are recorded in
.context/mcp-server-design.md section 12 (next commit) and the PR
body.

* mcp: record the real workerd smoke results in the design doc (#1323 review)

Section 12's phase 2 note is updated from "blocked" to the actual
result: all 14 checks PASS on real workerd via the mcp-smoke-entry.ts
throwaway entry, compatibility_date 2024-12-01 unchanged, with the
full PASS/FAIL table and per-call wall times. Also makes explicit
(section 10.2) that Bun installing two copies of zod 4 is expected,
not a defect -- the "two-copy risk" section 12 named ahead of time.

* mcp: fix contract scoping/citation edge cases; de-duplicate mirror literals (#1323 review A/B)

shared/contract/mcp.ts:
- Module doc corrected: the provenance envelope rides recording-level
  tool responses only (the section comment already said so; the
  top-of-file doc still claimed "every tool response").
- composeCitation: each DOI candidate is trimmed BEFORE the
  concept_doi/doi fallback, so a whitespace-only concept_doi correctly
  falls through instead of winning empty. Year requires exactly four
  digits (/^\d{4}$/), stricter than the ported Python's year.isdigit()
  (which also accepts a short prefix like "20") -- a converter-side
  follow-up to align dataset_citation itself, not scope here. New
  tests: whitespace concept_doi with a real doi keeps the DOI segment;
  created_at: "20" yields no year segment. nm000329 parity retested
  and still passes.
- Exports ZARR_STATUS_VALUES, ZARR_VERIFY_STATUS_VALUES,
  NEXT_CHEAPEST_TOOL_VALUES (the enums now build from them) and the
  SearchDatasetsHit type, so a mirror imports the literals instead of
  hand-duplicating them. searchDatasetsOutputSchema gains optional
  note/truncated.

backend/src/mcp/schemas.ts imports those arrays instead of redefining
the enum literals, and mirrors note/truncated. mcp-schema-parity.test.ts
extended: an unknown key nested inside results[] and inside cost_hint,
plus cases for note/truncated.

* mcp: degrade instead of throwing, attribute unresolved hits, drop casts (#1323 review C/D)

New backend/src/mcp/tool-types.ts: the shared ToolOutcome = { result,
metrics? } type both tools now return, replacing their two ad hoc
per-tool outcome interfaces (and the datasetId field on them -- server.ts's
withToolMetrics reads the dataset id from the request args instead, in
the next commit).

search-datasets.ts:
- executeDatasetSearch's method: "unavailable" (the FTS table missing)
  now answers isError: true naming the fallback (browse without query,
  or retry) instead of an empty page passed through silently, plus one
  console.error. envelope.warning carries through as `note`;
  envelope.truncated passes through too.
- A hit whose id has no row in the follow-up license/zarr map is no
  longer presented as license: null, has_zarr: false without comment --
  console.warn names the id, and a `note` on the response says which
  ids are unresolved. Extracted into a pure mergeHitsWithCatalog(results,
  extraMap), unit-tested (mcp-search-datasets.unit.test.ts) with a real,
  deliberately incomplete Map for the miss path -- no fake needed.
- The `r.has_hed as 0 | 1 | null | undefined` cast is gone: explicit
  === 1 / === 0 / else-null comparison instead. hits is typed
  SearchDatasetsHit[], not unknown[].

describe-dataset.ts: buildCostHint now takes the ALREADY-NARROWED
zarrStatus, never the raw column, so the cost_hint text and the
output's own zarr_status field cannot disagree. console.warn when a
raw zarr_status or zarr_verify_status is non-null but outside the
known set (zarr_verify_status in particular comes from free-form
sweep_stamps JSON with no DB enum behind it).

* mcp: type the tool name, add the outcome blob, attribute ids before the call (#1323 review E)

mcp-metrics.ts: McpToolCallEvent.tool is McpToolName (the contract's
closed set), not string. New outcome: "ok" | "tool_error" | "exception"
blob (4th position: tool, dataset_id, cache_status, outcome).
buildMcpDataPoint's doc and tests updated for the new field/blob count.
Design doc section 8 update lands in the design-doc commit, not here,
to keep that file's changes in one place.

server.ts: withToolMetrics reworked to
withToolMetrics(env, name, getDatasetId, fn) -- getDatasetId(args) runs
BEFORE fn, so a thrown D1 error is still attributed to the right
dataset instead of recording "-" because the tool never got far enough
to report one itself. fn now returns the shared ToolOutcome (previous
commit); try/catch records outcome: "exception" and rethrows;
result.isError records "tool_error", otherwise "ok".
search_datasets registers with `() => undefined` (it names no single
dataset); describe_dataset with `(args) => args.dataset_id`.

mcp-metrics.test.ts: outcome asserted throughout; new test drives a
genuinely unmigrated D1 (freshDb() with no schema) through the real
sub-app and asserts exactly one point records outcome: "exception"
with the dataset id taken from the request arguments -- PR #1323
review item G.13, landed here (not the later tests commit) since it is
the same behavior this commit introduces and splitting one file's
cohesive metrics-attribution tests across two commits would reduce
review clarity, not improve it; noted as a grouping deviation in the
PR body.

* mcp: one MCP-specific CORS header set, cite the design doc by content (#1323 review F)

New mcpCorsHeaders(origin): corsHeaders(origin) (zarr-data.ts) plus the
THREE fields that actually differ for this transport --
Access-Control-Allow-Methods (POST, OPTIONS), Access-Control-Allow-Headers
(the four Streamable HTTP request headers), and
Access-Control-Expose-Headers (MCP-Protocol-Version, Content-Type). The
OPTIONS preflight, the POST response re-wrap, the rate limiter's 429
branch, and app.onError all used to ship zarr's own Range/GET-HEAD-OPTIONS/
ETag-Content-Length-Content-Range-Accept-Ranges set on those four
responses; now they share one function and one comment explaining the
override, instead of the OPTIONS handler's own comment claiming only two
fields differed.

Module doc's "(decision 2)" citation replaced with the actual source:
.context/mcp-server-design.md section 3, "Host and routing". Grepped the
whole diff for "decision \d" and found one more real hit (design doc
section 10, from the earlier smoke-results commit) -- fixed there too.
The remaining "decision N" text in the design doc is phase 1's own,
already-merged prose citing ITS OWN plan by name ("decision 7 of the
phase 1 plan") -- a correctly-scoped historical citation, not the
dangling plan-number reference this item is about, so left as-is.

* mcp: fill the review's remaining test gaps (#1323 review G, items 12/14-20)

mcp-route.test.ts:
- G.12: a real in-memory CacheStorage stand-in (RateLimitCache, the same
  technique zarr-data-cache.test.ts's redirect-exemption block uses),
  ENVIRONMENT: "production", the ip bucket seeded at MAX_REQUESTS --
  asserts the 429 carries Access-Control-Allow-Origin for
  https://nemar.org and the MCP Access-Control-Allow-Methods.
- G.14: the converted row's citation contains "(v1.2.0)"; the pending
  row's (no dataset_versions row) contains no "(v" segment at all.
- G.15: explicit has_hed true/false on the two search_datasets hits,
  not just typeof.
- G.16: a self-contained EEG/MEG fixture pair (distinct modality, task,
  has_hed) proves each filter narrows the no-query path to the matching
  row, excluding a row that HAS a different value -- not just one with
  a NULL column.
- G.17: limit at the cap (100, echoes in output.limit) and limit: 1
  (exactly one row, output.count still the real total) on both the
  no-query and query paths.
- G.18: a non-matching query and has_zarr: true with no converted
  fixture both answer count: 0 with no isError.
- G.19: describe_dataset on a zarr_status: "failed" row (cost_hint
  names it); search_datasets(query) after DROP TABLE datasets_fts
  answers isError naming "search index" (the real degradation path,
  method: "unavailable", end to end).

blosc-decode.test.ts (G.20): every remaining guard triggered
individually against real, precisely-computed byte offsets in the
fixture (memcpyed flag, bit-shuffle flag, nbytes<=0 distinct from
blocksize<=0, nbytes % typesize, a block offset past the end, a zero
and an oversized compressedLen, and a decompressed-length mismatch --
the last built by declaring a smaller nbytes/blocksize than the real,
untouched zstd payload actually decompresses to).

G.13 (the thrown-D1-exception path) landed in the earlier metrics
commit instead of here -- see that commit's message for why.

* mcp: smoke script checks the D1 wipe's exit status, traps before spawning (#1323 review H)

The D1 state wipe no longer swallows a real failure with
`2>/dev/null || true`: it now only skips the wipe when the directory
is genuinely absent (first run), and exits loudly if find -delete
itself fails on an existing directory. trap cleanup EXIT is now
registered BEFORE wrangler dev is spawned (previously after), so an
interruption or failure between spawn and the old trap registration
point can no longer leave an orphaned --local wrangler/workerd process
holding the port and the D1 state lock.

Re-ran end to end twice after these changes: 14/14 PASS both times,
compatibility_date 2024-12-01 unchanged and starting the SDK cleanly.
Per-call wall times unchanged within noise (server/discover ~7-8ms,
tools/list ~3-4ms, tools/call ~2-7ms).

* mcp: design doc catches up with the review's implementation changes (#1323 review I)

Section 2: the dual-era legacy-serving claim now cites its two actual
sources by name -- mcp-route.test.ts's legacy test drives describe_dataset
in-process, mcp-smoke.sh's legacy check drives search_datasets under
real workerd -- rather than one generic "verified in phase 2" citation.

Section 5.1 (search_datasets): documents the three PR #1323 review
additions -- note (executeDatasetSearch's own warning, and unresolved
follow-up-lookup ids, joined when both fire), truncated
(executeDatasetSearch's own, query path only), and the search-index-
unavailable case now answering isError instead of a silent count: 0
(the ADR 0005 boundary: partial data still serves, an infra failure
does not get to look like "no matches").

Section 8: the outcome blob (4th position: "ok" | "tool_error" |
"exception") and withToolMetrics's getDatasetId(args)-before-fn
attribution fix, matching the reworked implementation.

Also fixes the "decision 9" plan-number citation in section 10
(introduced in an earlier commit of this same review pass) to describe
the decision by content instead.
…rview) (#1326)

* deps(backend): add hyparquet and fast-png for MCP phase 3

hyparquet reads events.parquet for get_events; fast-png encodes the
render_overview PNG. Both pure JS, no WASM, matching the fzstd/blosc
precedent from phase 2. hyparquet-compressors is deliberately NOT
added here -- see the get_events commit for why.

* feat(contract): add phase 3 recording-tool fields and legacy index schema

shared/contract/zarr-index.ts: zarrIndexLegacySchema, a deliberately
minimal lower bound for format_version 1/2 documents (no source_tree/
derived, source_commit unregexed since v1 predates the 40-hex
guarantee).

shared/contract/mcp.ts: additive fields on list_recordings (nullable
source_commit, index_format_version, discovered/failure/pending_count,
excluded_legacy_non_raw_count, note, envelope; group summary gains
n_view_levels/n_samples/view_chunk_columns) and get_events
(limit/offset, total_count/truncated/note/envelope) and render_overview
(columns_read/chunks_read/bytes_read/envelope). All additive and
passthrough, per the repo's lower-bound convention.

* refactor(mcp): extract the shared public-catalog row read (decision 3)

catalog-row.ts pulls describe_dataset's D1 query, row shape, and
not-found error text out into loadPublicDatasetRow/PublicDatasetRow/
datasetNotFoundResult, plus a new isZarrReady/zarrNotReadyResult pair
for the recording-level tools. describe_dataset now calls it instead
of owning the query itself; its existing tests (mcp-route.test.ts)
stay green unchanged, proving the extraction is behavior-preserving.

* feat(mcp): add the index.json reader and the projection cache

index-reader.ts (decision 2): fetches index.json IN PROCESS through
the real zarr sub-app (D1 gate, edge cache, purge list unchanged),
narrows format_version 3 against zarrIndexSchema and everything else
against zarrIndexLegacySchema, reports sourceCommit: null rather than
crashing on a non-40-hex source_commit.

projection-cache.ts (decision 4): synthetic
https://mcp.nemar.org/_cache/<id>/<commit>/<projection> keys, 7-day
TTL, the safeCachePut discipline from zarr-data.ts (a throwing
cache.match/put is logged once and treated as a miss/no-op).

zarr-fidelity-sweep.ts: exports GITHUB_RAW_ORIGIN and rawContentUrl
(no behavior change) for get_events' events.tsv fallback to reuse.

Test infra: InMemoryCache/keyFor moved to test/helpers/cache.ts (was
private to zarr-data-cache.test.ts); a new Bun.serve() fixture-server
helper stands in for both S3 and raw.githubusercontent.com; fixtures
under test/fixtures/mcp/ are live captures (curl -A "nemar-cli/mcp-
phase3") of nm000329 (v3, 112 stores, events.parquet, one view-5 and
one view-4 pyramid chunk) and nm000111 (v1), plus a synthetic
events.tsv with n/a cells and an HED column.

* feat(mcp): add list_recordings tool (decision 6)

tool-types.ts: RecordingToolDeps, the shared dep bundle (env,
executionCtx, cache, fetch, zarrRoutes, rawGithubBase) the three
recording-level tools all take.

envelope.ts (decision 5): buildEnvelopeForStore, the one place that
infers a legacy (v1) store's source_tree/derived/engine_version and
attaches the fixed legacy note, so the three tools cannot diverge.
Takes resolved index-level facts as plain fields (not a whole parsed
index document) so an envelope can be built on a projection cache HIT
without a second index.json fetch.

list-recordings.ts: the tool itself, plus loadRecordingsProjection/
resolveRecording/recordingNotFoundResult, exported for get_events and
render_overview to share -- a list_recordings call primes the same
cache entry those two read, and vice versa. Excludes a legacy store
under derivatives/sourcedata/code (ADR 0027) and counts the exclusion.

Not yet wired into server.ts; lands with the mirrors/wiring commit.

* feat(mcp): add get_events tool (decision 7)

Primary path: reads events.parquet once per (dataset_id,
source_commit) with hyparquet, groups rows by store_path, writes
every store's rows plus a store-list summary to the projection cache
in one pass. Converts hyparquet's BigInt sample_index to Number once.
No columns filter, so subject/session/task/run and any x_-prefixed
extra pass through per eventRowSchema's .passthrough().

Fallback: derives the sibling events.tsv from the recording's path
(BIDS naming) and fetches it from raw.githubusercontent.com via
zarr-fidelity-sweep.ts's rawContentUrl; computes sample_index locally
(Math.floor(onset_s * rate + 0.5), the converter's own formula) and
flags estimated: true. A clean 404 answers events: [] with a note.

Codec: a hand-rolled ZSTD-only compressors map built on fzstd, NOT the
hyparquet-compressors package -- that package's compressors export
eagerly WASM-compiles its SNAPPY entry (hysnappy) at module load,
which crashes isolate startup under real workerd regardless of
whether any dataset's parquet ever uses SNAPPY (none does; every
column sampled is ZSTD). Found via the smoke script, not bun test.

Not yet wired into server.ts; lands with the mirrors/wiring commit.

* feat(mcp): add render_overview tool (decision 8, moved from phase 4)

overview.ts: pure helpers, no I/O. pickViewLevel returns the COARSEST
view level satisfying width_px (biosigio's Math.floor(n/4) per level,
applied iteratively); buildChunkPlan computes chunk keys from
view_chunk_columns (default 1024) with no special-casing for "the
last level" -- the ceil() formula already produces one chunk when a
level's own column count is under the chunk size, verified against
nm000329's real level 4 (541 cols) and level 5 (135 cols, one chunk of
chunk_columns: 135); reassembleViewChunks stitches decoded [2,
n_channels, chunkCols] chunks into one [2, n_channels, totalCols]
buffer; renderOverviewPng draws one grayscale band per channel
(rowPx = clamp(floor(1200/n_channels), 2, 24)), a min-max bar per
width_px column bucket, via fast-png.

render-overview.ts: resolves recording/group off the shared
recordings projection (never a probe fetch, never zarr.json), fetches
only the chosen level's chunks straight from data_base, decodes with
decodeBloscZstdInt16, caches the whole PNG per (dataset_id,
source_commit, recording, group, width_px) -- a hit skips fetch,
decode, and encode entirely.

Not yet wired into server.ts; lands with the mirrors/wiring commit.

* feat(mcp): wire the three phase 3 tools with zod4 mirrors (decisions 9, 10)

schemas.ts: zod4 mirrors for list_recordings/get_events/render_overview
input+output, plus a provenanceEnvelopeSchema4 mirror (including the
ADR 0028 sss-iff-derived superRefine) since it rides all three outputs.

server.ts: BuildMcpServerDeps grows cache/fetch/zarrRoutes/rawGithubBase;
a RecordingToolDeps bundle is built once per request and passed to all
three new registerTool calls.

routes/mcp.ts: McpRoutesDeps grows the same four fields, defaulting to
caches.default (lazily), the global fetch, the real zarrDataRoutes
instance, and GITHUB_RAW_ORIGIN -- threaded into buildMcpServer.

Tests: mcp-recording-tools.test.ts (list_recordings + get_events
through the real route: cache-hit-skips-index.json proof, v1 legacy
listing, the events.tsv fallback's n/a-as-null and floor(onset*rate+0.5)
sample_index, unknown-recording error text) and mcp-overview.test.ts
(pickViewLevel/buildChunkPlan/renderOverviewPng unit tests against the
real decoded nm000329 fixture chunk, plus the route-level request-log
proof that render_overview reads only view/<L>/ keys, never /0/c/ or
zarr.json, and that a repeat call is a cache hit with an empty log).
mcp-route.test.ts's tools/list assertion now expects all five tools.
mcp-schema-parity.test.ts gains the decision-10 case table (defaults,
caps, cap+1, malformed id, width_px/limit boundaries, negative offset)
for all three new schema pairs.

* test(mcp): smoke-test five tools under real workerd

tools/list assertion now checks for all five tool names; a new check
calls list_recordings(xx000000) against the migrated-but-empty local
D1 and expects the shared not-found error, mirroring the existing
describe_dataset(xx000000) check.

This script is what caught the hyparquet-compressors/hysnappy
WASM-under-workerd crash during implementation (bun test alone stayed
green -- see the get_events commit for the fix). Two consecutive runs
here, all 15 checks PASS, compatibility_date unchanged at 2024-12-01.

* docs: update MCP design doc for phase 3

Sections 5.3-5.5: the v1/legacy behavior, get_events pagination and
the ZSTD-only codec finding, render_overview's level rule and image
geometry, all against the live nm000329/nm000111 measurements.
Section 7: the concrete recordings/events/overview cache key shapes
and the 7-day TTL. Section 9: hyparquet/hyparquet-compressors/fast-png
rows updated with the real workerd verdicts. Section 10.3 (new): the
measured phase 3 bundle delta. Section 12: render_overview marked DONE
under phase 3 (decision 1's move from phase 4), the zarrita open item
closed.

* fix(mcp): correctness fixes from phase 3 review (group A)

Addresses PR #1326 review items 1-13:

- CRITICAL: thread sss/units_report through list_recordings' projection,
  get_events, and render_overview's envelope call sites -- previously
  dropped, crashing buildEnvelopeForStore's SSS-iff-derived assertion
  (ADR 0028) on any MEG SSS derived store (on003392).
- CRITICAL: get_events resolves the requested group name FIRST and
  returns an explicit groupNotFoundResult error (shared with
  render_overview) instead of silently returning an empty row list for a
  typo'd group.
- Decouple get_events' parquet DATA-SOURCE gate (eventsParquetUrl alone)
  from its CACHE gate (sourceCommit !== null); the two were wrongly
  combined, causing a spurious fallback to the strictly-worse TSV path.
  Wrap the parquet read in try/catch with a dedicated error result.
- Write "[]" cache placeholders for every recordings-projection store
  absent from a dataset's real events.parquet content.
- loadEventsFromTsvFallback returns a discriminated union: only a 404 is
  absence; any other non-2xx or a thrown fetch is a distinct error result
  (tsvFetchFailedResult), not silently treated as "no events". Track
  bytesFetched for the TSV path too.
- Log tool exceptions (console.error) in withToolMetrics before rethrow.
- Move narrowZarrVerifyStatus out of describe-dataset.ts into
  catalog-row.ts (exported) so envelope.ts can use it instead of an
  unsafe type cast on the raw D1 column.
- render_overview's envelope dtype is "int16" only on a cache MISS (the
  call that actually decoded a view array); a cache HIT now reports
  dtype: null instead of a hardcoded, misleading "int16".
  wired through NO_COMMIT_NOTE, mirroring list_recordings exactly.
- Version the projection cache payload shape (PROJECTION_SCHEMA_VERSION
  in the cache key) and validate a cache hit against a zod schema before
  trusting it; a schema mismatch is a miss, not a crash.
  guard with a descriptive throw.
- list_recordings: note a zero-match modality filter against the
  dataset's actual known modalities; updated tool description to call
  out that a count of 0 is not an error.

No mocks: all coverage lands in group B against real D1, a real
Bun.serve() fixture upstream, and a real in-memory cache.

* test(mcp): coverage for phase 3 review findings (group B)

Addresses PR #1326 review items 14-24:

- pickViewLevel: add boundary cases (135, 2167 columns) to the level
  table.
- reassembleViewChunks: new pure-function test with three literal
  synthetic chunks (a short last one), spot-checking column values at
  every boundary.
- A constant channel (min === max) renders without NaN, one stable band.
- New MEG SSS derived-store coverage (item 1's crash, now with real
  tests): real on003392 MEG SSS index data with geometry overridden to a
  small synthetic size, plus matching synthetic blosc-encoded chunk
  bytes generated via scripts/zarr/generate_mcp_test_fixture.py (same
  numcodecs Blosc config biosigio's real exporter uses) -- covers
  list_recordings, get_events, and render_overview against a real SSS
  store.
- get_events edge cases: offset past end, limit 5000, a nonexistent
  group (item 2's fix), a CRLF events.tsv with half-sample rounding, a
  parquet file with zero rows for the requested store.
- modality + min_duration_s combined filtering.
- zarr_store_count 0 answers the not-ready error, for all three tools.
- A legacy v1 index with an empty source_commit: the projection cache is
  never even consulted (item 17) -- proven with a SpyCache wrapper
  around InMemoryCache that records every URL passed to match(), since
  the zarr sub-app's own independent edge cache made a naive
  upstream-fetch-count assertion give a false pass.
- projection-cache: a stored entry that fails schema validation is a
  miss, not a crash (item 11's schema versioning).
- metrics: cache_status ("miss" then "hit") for all three recording-level
  tools, driven through a real fixture server + zarr routes.
- schema parity: provenanceEnvelopeSchema/schema4 (5 cases, including
  both invalid sss/derived combinations) and the envelope embedded in
  all three tools' output schemas.
- fixture-server: malformed Range ("bytes=abc-def") and out-of-range
  Range both answer 416 + Content-Range: bytes */<len>.

No mocks: real D1 (bun:sqlite), a real Bun.serve() fixture upstream, a
real in-memory projection cache.

* docs(mcp): cite by content, not plan decision numbers (group C)

Addresses PR #1326 review items 25-34:

- Remove every "decision N" / "plan decision N" citation from code,
  tests, helpers, shared/contract/mcp.ts, and
  .context/mcp-server-design.md, per AGENTS.md's citation-discipline
  rule. Replaced with ADR numbers, design-doc section references, or
  plain descriptions. (zarr-fidelity-sweep.ts keeps its OTHER "decision
  N" comments -- those cite epic #1181's own plan, not this one, and are
  out of scope here; only the single line this PR's diff touches was
  fixed.)
- Stage ADR 0050's content in the design doc's section 12: the "no WASM
  in the Worker bundle" rule, covering both numcodecs (phase 1) and
  hysnappy/hyparquet-compressors (phase 3), plus the real reason the ADR
  file itself is deferred (ADR 0049 not yet on dev; the ADR index test
  enforces gapless numbering) rather than "the plan's numbering rule".
- Add a phase 3 smoke-results table (15 checks, five tools) to section
  12, mirroring phase 2's own table -- both from two consecutive live
  runs of backend/scripts/mcp-smoke.sh.
- Fix overview.ts's computeViewLevelColumns doc comment: the claim that
  the iterative Math.floor(n/4) loop differs from Math.floor(n0/4**L) in
  one step was false for non-negative integers (floor-division
  associativity); reworded to say the loop mirrors biosigio's own
  implementation rather than claiming a numerical difference that does
  not exist.
- Fix projection-cache.ts's module doc: correct an inverted claim about
  which documents get the short untokened TTL split (zarr.json /
  dataset-level docs) versus the flat 24h chunk TTL this file's
  projections share the rationale with.
  index-reader.ts: DEFAULT_ZARR_BASE's fallback behavior is NOT shared
  with zarrPurgeTargets() (empty list when unset) or s3PublicUrl
  (bucket-URL fallback); clarify it is a new, distinct fallback.
- zarr-fidelity-sweep.ts: rawContentUrl's doc pointed at .context/README.md
  for the "nemarDatasets names a dataset's repo after its own id" fact;
  it actually lives in .context/systems-inventory.md.
- fixture-server.ts: clarify FixtureRequestLogEntry.url differs in NAME
  (not just shape) from zarr-data-cache.test.ts's LoggedRequest.path.
- cache.ts: restore the full 206-refusal comment (with its #1181 review
  item 13 citation) instead of a bare "unchanged" claim.
- Add "(as of 2026-09-08)" to the "about half the catalog is v1" claim,
  in both index-reader.ts and the design doc, with a cross-reference
  between the two.

* docs: drop dangling plan-decision citations

Comments citing "decision N" point at epic plan briefs that do not exist
in the repo. Each comment already states the rule it cites, so the
parenthetical is removed rather than rewritten. Comment-only.

* fix(mcp): bound the get_events per-store cache fan-out

A parquet miss wrote one cache entry per store in the dataset. nm000281
publishes 25,253 stores, so a single get_events call could issue 25k
Response constructions and JSON.stringify calls inside one request's
waitUntil. Cap the fan-out at 2000 stores; the requested store's entry
and the _stores summary are always written.
* feat: add read_window taste caps and fields

* feat: add read_window's pure decode helpers

* test: add synthetic sharded store fixture

* feat: implement and register read_window tool

* test: add read_window route-level tests

* test: extend mcp parity/contract test tables

* chore: smoke-test and measure read_window

* docs: update MCP design doc for phase 4

* fix(mcp): shard footers carry a constant entry count

A shard index always has shard_samples / chunk_samples entries, for every
shard including the truncated last one; chunks past the array's extent are
absent entries, not omitted ones. nInnerForShard computed the count from the
remaining extent instead, so for nm000329's shard 1 it read the last 1028
footer bytes rather than 1204, starting 11 entries into the real footer.
Every chunk served from that shard was 11,000 samples (44 s) late, and
nothing errored because each misread entry pointed at a real, full-size,
cleanly decoding chunk: sample 75,000 returned sample 86,000.

A present chunk that straddles n_samples is also stored full size and
fill-padded, so a decoded chunk's stride and its valid span are separate
quantities; conflating them made the length check reject the tail of every
recording whose length is not a multiple of chunk_samples.

The synthetic generator had encoded the same wrong rule, which is why the
suite was green. It now derives the entry count from the chunk grid, pads the
straddling chunk, and emits its expected values as data. The real boundary
footer (c/0/1) is committed as a fixture so the rule is pinned against the
actual producer rather than against our own generator.

Also from the review: report filled ranges instead of returning fill values
as if they were signal, refuse a Range request answered with a bare 200,
refuse an empty channels array before any byte is fetched, make
ShardIndexEntry and ChunkRead discriminated unions, and reject a window that
rounds to zero samples.

* test(mcp): cover the boundary shard, caps, and refusal paths

Every one of these was a gap a mutation proved: the code could be broken and
the suite stayed green.

- The channel-samples cap, the only taste bound no schema enforces, was
  tested ~14x past its limit, so > versus >= was indistinguishable. Asserted
  now at exactly 65,536 and 65,537 through the pure predicate, plus a
  product-versus-sum case.
- An empty channels array with taste: true.
- Unknown recording, unknown group, a group with no sharding geometry (which
  refuses a taste but still serves a recipe), and a group with no usable
  rate. No v3 fixture in the repo could reach the last two, so both index
  variants are built by removing fields rather than committing near-duplicates.
- A Range-ignoring origin, using the fixture server's new opt-in capability.
  The assertion names the protocol reason: tolerating a bare 200 still trips
  the byte-length check further down, so a looser assertion passed with the
  status check removed.
- Expected physical values now come from the generator's emitted
  nm099500-expected.json instead of a hand-written mirror of its formula, and
  a sample in the boundary chunk's fill padding is proved to be refused
  rather than served.

* docs(mcp): correct the shard geometry and byte accounting

Section 5.6 stated Zarr's "last chunk may be shorter" rule for the shard
index and for a straddling inner chunk. Neither holds: the entry count is
constant per shard and a straddling chunk is stored full size with fill
padding. Both are now recorded with the measured evidence, including what the
misread returned from live data.

Also: bytes_read was described as Range-read bytes in two places while also
counting the array-metadata GET, which is a plain full GET; on003392's
shard_samples is named per store (68000 on sub-06, 69000 on sub-01) rather
than presented as one figure for the group name; and the memory table gains a
boundary-shard case, since the first two cases both start at sample 0 and so
never touched the shard the defect lived in.

* test(zarr): make the format claims executable, not prose

Phase 4's shard-footer defect shipped because a claim about the on-disk
format was measured on two stores, written into the design doc, and then
trusted by the code, the tests, the fixture and five reviewers alike.
Everything agreed with the prose; the prose disagreed with production.

Two gates so that class of bug has to announce itself:

zarr-geometry-conformance.ts (bun run zarr:geometry-check) checks every
invariant the readers depend on against every index the catalog publishes:
footer entry-count divisibility, the index agreeing with the array it
describes, inner chunks spanning every channel, the codec chain and dtype,
per-channel scale/offset lengths, and every events.parquet column being ZSTD
with the columns eventRowSchema requires. It also reports the observed
distributions, since a small sample is what makes a varying figure look
constant. A transient 429/5xx is retried, reported separately, and never
counted as a violation; concurrency is 4 after an earlier 12 drew 500s from
the host that serves real users.

zarr-schema-contract.test.ts now requires the published JSON Schema and the
zod contract to agree on required-ness field by field, driven off the shared
fixture, with the deliberate asymmetry (the producer's gate refuses an
undeclared field, consumers tolerate it) asserted as a difference.

First full run: 314 v3 indexes, 90,966 groups, 285 parquet footers, 13,023
columns, zero violations. It also corrected two claims this repo was
repeating: chunk_samples takes 33 distinct values and shard_samples 256, not
the single figures a two-store sample suggested, and the widest store is 415
channels, not 320.
* docs: ADR 0049 supersedes 0025 on compute locality

Compute runs in the browser by default with eegprep as the engine, OSA owns the execution runtime, and only HPC submission is gated. Restates the broker decision from 0025 and marks 0025 superseded. Index test: 7 of 8 pass; the gapless-numbering check fails until epic #1272 lands ADR 0047 and 0048 on dev.

* docs: ADR 0049 corrects the picard dependency claim

python-picard is pure Python with an optional numexpr fast path; the browser blockers in eegprep's declared dependencies are oct2py, pyedflib, and psutil. Notes the no-BLAS numpy build in Pyodide and the torch-free ICLabel need.
* docs: file ADR 0050 and add a Python client verification

ADR 0050 formalizes the rule two phases learned the hard way: no WebAssembly
in the Worker bundle, ever, for any package this server depends on. Both
failures (numcodecs instantiating WASM from a runtime fetch, and
hyparquet-compressors compiling hysnappy's WASM at module load and so crashing
isolate startup for every request) were invisible to bun test and caught only
under real workerd, which is why the smoke script is the gate rather than an
optimization. It could not be filed until ADR 0049 landed on dev, since the
index test enforces gapless numbering; the content staged in the design doc is
now a pointer to the ADR.

scripts/mcp/verify_streamable_http.py drives a target host with the official
Python MCP SDK: the negotiated revision, all six tools with their cache hint,
then a chain from search through describe and list_recordings into the three
recording tools and both read_window modes, against a recording it discovers
rather than one hardcoded here. mcp-smoke.sh checks our own framing with curl
and says nothing about whether a real SDK can talk to us, and OSA speaks this
SDK, so this is the client that has to work. It is also where the docs page
will copy its examples from.

Its SDK usage was validated against a local MCPServer harness rather than
discovered during a cutover, which caught two things worth having: any check
failure inside the client's task group came back as a bare "ExceptionGroup:
unhandled errors" with none of the message that explains it, and an over-cap
refusal can arrive either as a tool error or as a raised protocol error, so the
check accepts both.

Design doc records the verified SDK version and the two assumptions in #1297
that turned out to be wrong: there is no For-agents guide to extend, and OSA
has no MCP runtime at all while its two existing NEMAR tools call an endpoint
that now 404s.

* ci: run the ADR index test when ADRs change

test/adr-index.unit.test.ts validates gapless numbering, README index
agreement, and superseded-target existence. Nothing in the path filters
matched .context/**, so it never ran when those files changed: #1292 and #1330
both showed a fully green check list with the one test that could have failed
skipped, and both were verified by hand instead.

Same shape as the shared/** gap this file already documents, same fix. Wired
into unit-pure, with a self-reference so editing the filter re-runs it.

* fix: harden the MCP verification script

Review of scripts/mcp/verify_streamable_http.py found the script could not
connect at all and could report a pass it had not earned. Fixed:

- The default URL omitted the /mcp transport path. GET / is a descriptor and
  POST / answers Not Found, which during a cutover reads like an
  unprovisioned hostname. A bare host is now normalized and reported.
- The over-cap probe wrapped its call in `except Exception`, so a 429 or a
  502 became a PASS. Removed: both a tool error and a schema rejection
  arrive as is_error, so an exception there is a transport failure.
- The probe sat exactly AT all three caps, which the schema accepts (`>`).
  Now one second over the duration cap with one channel, so exactly one cap
  is crossed, and the refusal text must name it.
- The envelope check tested 3 of 14 required fields. Now all 14, plus
  non-null where the contract forbids null, and required only where the
  contract makes it required: it is .optional() on three tools, so failing
  hard everywhere reported missing provenance against a valid legacy index.
- The subject was `results[0]`, now scanned for a v3 index with a usable
  group. has_zarr means converted, not v3, so a legacy hit is legitimate.
- Read `results` only; `datasets` is catalog.json's field.
- Notes are counted and repeated in the closing line, so a degraded run
  cannot look clean.

Adds the wire-level test the script's cap assertion depends on: an over-cap
taste answers isError with a text block naming the cap and the recipe
remedy. Validated against a local MCPServer stand-in with twelve fault
injections; all ten defects fail with the right message and the two honest
degradations exit 0 with a note.

* fix: correct ADR 0050's platform claim and the docs target

An accuracy review of ADR 0050 and the design doc's phase 5 entry found four
wrong claims, two of which would misdirect future work. All verified before
changing, two by re-running the experiment.

ADR 0050 said there is no flag permitting dynamic WASM compilation, so the
restriction was the platform's and not configurable. Inverted. Measured under
wrangler dev --local: compiling at MODULE LOAD is permitted by
allow_eval_during_startup, which is the default from compatibility_date
2025-06-01, so the phase 3 shape is blocked only because this worker pins
2024-12-01. Compiling inside a request stays disallowed under every
combination tested. So the decision survives but gains a real consequence: a
routine compatibility_date bump silently removes the guard that caught the
phase 3 failure, which is why the rule is written down and smoke-gated rather
than left to the runtime. The flag's docs never mention WebAssembly, so the
ADR now says the behavior is measured, not documented.

ADR 0050 also framed both failures as dependencies that broke the server with
a green suite. Neither was ever in backend/package.json: numcodecs was the
spike's path (a), and hyparquet-compressors was removed before merge. The
lesson holds, the history does not, and an ADR outranks the design doc.

Design doc: the For-agents guide #1297 names does exist, published at
docs platform/for-agents.md since 2026-09-03, so the docs work is a section
on that page and #1297's premise was right. develop/zarr-contract.md is gone,
split into six pages under platform/zarr/. Both errors came from reading a
stale local checkout instead of origin/main.

Also: #1324 has three non-handler exports, not two strings; the fixtures are
real bytes where a real store exercises the path and synthetic only where the
catalog has no such shape; the subpath-import option is named; and the
custom-domain zone-edit caveat is moved next to the sequencing it gates.
Two critical findings from the epic-level review, both confirmed by running
the code rather than reading it.

render_overview threw a RangeError on its DEFAULT path for much of the
catalog. reassembleViewChunks derived each chunk's width from its decoded
length and documented "the last chunk may be shorter", which is false: Zarr
stores a boundary chunk FULL SIZE and fill-padded. Verified against
nm000329 eeg_250hz level 3, which width_px 800 selects: all three real
objects carry blosc nbytes 258048 = 2 x 63 x 1024 int16, so the third holds
1024 columns of which 119 are real. The old code advanced past the end,
wrote each channel over the next channel's region, then overran the buffer.
This is the same nominal-stride versus valid-span split sharding.ts already
documents for level 0; the trap had not been carried to the view path.

chunkColumns is now passed as the stride and clamped to the level's own
columns, matching the real chunk grids read from S3 (view/4 and view/5 clamp
to 541 and 135). A short chunk is now refused instead of inferred from. The
reassembly moved inside the try, so a geometry disagreement is this file's
typed tool error rather than an opaque JSON-RPC internal error.

The synthetic fixture encoded the same wrong assumption, widths (50, 50, 17),
which is why the suite was green; regenerated full-stride with fill padding.

Second: /mcp matches none of the rate limiter's path patterns, so a caller
sending any 32-character bearer selected the token bucket, on a surface with
no authentication at all. That gave a fresh 1000/min allowance per rotated
string, bypassing the 500/min IP floor, and drove a D1 tokens-to-users
lookup per novel token from one anonymous IP. Adds an explicit
anonymousSurface option and passes it from the MCP sub-app, so the whole
sub-app including the descriptor keys on IP. Tested through the real
sub-app, not the helper: both tests fail when the flag is removed.
The cache layer rested on a claim that is false: "a re-conversion mints a new
source_commit, so the OLD key never again resolves to different bytes".
source_commit is the dataset repo's HEAD, and the documented back-catalog
mechanism re-converts at an UNCHANGED HEAD, an engine bump re-queues a done
row and bumps no dataset version (ADR 0033); a --clean rebuild and a retry
after an infra failure are the same shape.

So a re-conversion left every MCP projection stale for a 7-day TTL while
index.json itself refreshed in 5 minutes, and nothing purges these synthetic
keys. Two of the namespaces make that a correctness bug, not staleness:

- array/ caches scale[] and offset[], which a taste multiplies by. Stale
  constants against re-quantized data return silently wrong PHYSICAL values,
  with no error and nothing in filled_ranges.
- shardidx/ caches parsed byte offsets and replays them as Range reads
  against whatever object is at that key today. Every 206 and length check
  passes by construction, because the origin returns exactly the bytes asked
  for, just the wrong ones.

The key now carries datasets.zarr_converted_at, set to datetime('now') by
every conversion and readable from the D1 row the tools already load before
any fetch, which is what rules out the index's ETag or updated_utc: the whole
point of the recordings entry is to answer without fetching index.json.

The namespace is also environment-derived now. It was the literal
mcp.nemar.org, but caches.default is zone-scoped and both MCP hostnames are
custom domains on the nemar.org zone, so prod and staging read and wrote each
other's entries while their data_base pointed at different buckets.

Also bounds render_overview, whose cache key was the caller's raw width_px: a
caller walking 1..4000 forced up to 4000 renders and cache writes per
recording and group, each re-fetching the same view chunks from S3 with no
edge cache in front. The key now carries the level and a width rounded UP to
one of nine buckets, floored at the level's own column count so it stays a
pure downsample. Level selection deliberately keeps using the RAW width:
choosing from the rounded value read a FINER level than asked for. And the
chunk plan is capped at 64, since the count comes from the index and a
shallow pyramid on a long recording could fan one call out past Cloudflare's
1000-subrequest ceiling.

Tests: a re-conversion at an unchanged commit no longer serves the old entry,
prod and staging keys differ, and the fixtures carry a conversion stamp.
Both are on a free, anonymous endpoint, so one request could cost arbitrary
Worker memory and S3 bandwidth.

get_events read every events.parquet WHOLE, before applying limit/offset.
Measured against the live bucket: 285 public datasets publish one, 13 over
5 MB, and nm000104 is 99,863,763 bytes with 5,411,570 rows across 1131
stores. So get_events(nm000104, limit: 1) pulled 95 MB through the Worker,
zstd-decompressed it in pure JS, materialized 5.4 M row objects and ran 5.4 M
zod parses. That does not fit a 128 MB isolate, and the failure re-amplified:
the isolate died before waitUntil ran, so nothing cached, so the next call
repeated the whole read, at up to 500 requests a minute from one IP.

Now bounded at 16 MiB / 100,000 rows, checked from the footer this path
already fetches, so a refusal costs one footer read. Refusing beats paging
because a row's store_path is only known after the row is read, and it costs
the caller little: index.json publishes events_parquet as a public URL, so
the refusal hands that over instead. That is the recipe-first posture ADR
0049 sets for signal bytes.

read_window's four taste caps all bound the RESPONSE and none bound the
DECODE, because a stored chunk holds every channel in the store: cost is
n_channels(store) x windowSamples no matter how few channels were asked for.
sharding.ts says so in prose and no guard was added. The worst legal request
in the catalog, found by scanning all 108 meg/ieeg/emg indexes, is on004696's
ieeg_1000hz: 256 channels at 1000 Hz, and {channels: [0], duration_s: 60}
passes every cap then decodes 15.4 M int16 samples, about 31 MB retained, to
return 500 KB. The 415-channel store in the archive makes it about 50 MB, and
128 MB is per ISOLATE, so two concurrent tastes take out every unrelated
request in flight. Now bounded at 4 M decoded samples, refused before any
fetch, with a message naming the STORE's channel count and a duration that
does fit, because "ask for fewer channels" is the obvious next move and the
one that does not work.

Fixtures: a real 100,001-row parquet, 4.5 KB because every column is a
repeated constant, and a v3 index with on004696's geometry. Both guards
mutation-checked, and both route tests assert nothing was fetched.
The zod3/zod4 drift test only compared whether the two copies AGREED, so a
case both rejected passed silently. The read_window taste fixture was invalid
(no filled_ranges, required on both copies), which made its own case vacuous
and the three "field removed, still rejected" cases built from it vacuous too,
since the base was already rejected. One was labeled "accepted" while both
copies rejected it. Net effect: phase 4's entire taste output contract,
filled_ranges' nested shape included, was unguarded by the test the epic
relies on for exactly that.

The fixture now carries a non-empty filled_ranges, and assertParity asserts
the VERDICT, taken from the label by the file's own convention. That turns
every "-- rejected" label into a real assertion as well. Adds the missing
cases: filled_ranges absent, an entry missing end_s, and an empty array.

Separately, envelope.dtype meant three things. The contract says it comes
from the array-metadata document, but render_overview never reads zarr.json
(its own module doc says so) and emitted "int16" on a cache miss and null on
a hit, while read_window emits the real value including on a cache hit. It is
now defined as the level-0 array's dtype from that array's own zarr.json,
null when the answering tool did not read that document, and it explicitly
does NOT mean "read on this call" -- chunks_read and bytes_read are for that.
So only read_window populates it and render_overview always sends null.
get_events answered dishonestly in three ways.

Its "no rows" note was computed on the PRE-filter rows, so a group that
exists and has no events answered events: [], total_count: 0, note: null,
indistinguishable from a confident "this group has no events". The note is
now computed after filtering and names the group it was empty for.

It also did not filter by group at all unless the caller named one, while
events.parquet is one row per (event, CHANNEL GROUP). A two-group store
therefore returned every event twice and reported total_count as double, with
no note, while the envelope described only the first group's rates. It now
defaults to the first group like render_overview and read_window already do,
and says which group answered and what else the recording has. A new
two-group fixture makes the doubling visible: the test reads 4 where it used
to read 8.

And the dropped-row count lived only in the miss path's return value, so it
reached the one caller that happened to populate the cache entry; every later
call was a hit reporting 0 dropped rows with a total_count that quietly
omitted them. The count now travels in the entry (schema version bumped to 2),
with a fixture carrying one row eventRowSchema rejects.

The tsv fallback's no-rate note now says the file was NOT FETCHED, rather than
leaving an empty list that reads as "this recording has no events". Its byte
accounting falls back to the body length when content-length is absent, the
same thing array-metadata.ts already did, so the figure that would reveal an
oversized read cannot silently report 0.

index.json is bounded at 24 MiB for an inline parse. Its size is not bounded
by anything this code controls: nm000281 publishes 25,253 stores in
12,846,915 bytes, and describe_dataset's own tool description already cites
that number as the reason it never reads the file, while the three tools that
do read it were unbounded. The bound clears today's largest with headroom,
refuses without reading the body when the header is over it (proven by making
the body throw if read), and hands over the public URL.

read_window's data-driven throws are typed results now. An index whose
shard_samples is not an exact multiple of chunk_samples used to escape as an
opaque JSON-RPC internal error; it is a fact about someone else's published
geometry, so it is a typed refusal, computed before any I/O since the index
alone determines it. Same for planShardReads. And a cached shard footer whose
entry count disagrees with the geometry is treated as a miss rather than
handed on to throw.

Removed the events/_stores cache entry: written once per parquet miss,
validated by its own schema, read by nothing but a test.
@neuromechanist
neuromechanist merged commit 98b364c into main Sep 9, 2026
28 checks passed

This branch was previously deployed

2 inactive deployments
development — 60f835b2 Deployed Sep 9, 2026 by neuromechanist via deploy-dev #886
npm — 60f835b2 Deployed Sep 9, 2026 by neuromechanist via publish #2435
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant